fix(#159): retry fullsend post-review on transient API failures#161
fix(#159): retry fullsend post-review on transient API failures#161fullsend-ai-coder[bot] wants to merge 3 commits into
Conversation
|
/fs-fix rebase and resolve the conficts. |
|
🤖 Finished Fix · ✅ Success · Started 11:31 AM UTC · Completed 11:37 AM UTC |
Wrap the fullsend post-review call in a retry loop (max 3 attempts with 5s/15s backoff) to handle transient GitHub API errors such as 422 Unprocessable Entity during PR transitional states. Previously, a single non-zero exit (other than code 10 for stale-head) caused post-review.sh to exit immediately. This meant a valid review comment could be posted while the formal review API call failed, leaving the PR without an outcome label and requiring human diagnosis. Changes: - scripts/post-review.sh: add retry loop around fullsend post-review. Exit code 10 (stale-head) bypasses retry and uses existing handler. On exhausted retries, a degraded-mode fallback applies the outcome label directly so the PR is not left in limbo. The script still exits 1 to signal the formal review was not posted. - scripts/post-review-test.sh: add 7 integration tests covering retry-then-succeed, retries-exhausted with fallback for approve/ comment/request-changes actions, stale-head bypass, and retry log message format. POST_REVIEW_RETRY_DELAY env var overrides backoff for all retries (set to 0 in tests to skip sleep). Closes #159
3327064 to
313ef13
Compare
🔧 Fix agent — iteration 1 (human-triggered)Rebased onto origin/main, resolved conflict in post-review-test.sh by merging both draft PR tests and retry tests, and updated degraded-mode fallback to include PR_IS_DRAFT check for consistency with main's outcome-label logic. All 58 tests pass. Fixed (1):
Tests: passed Decision points
Updated by fullsend fix agent |
|
/fs-review |
|
🤖 Finished Review · ✅ Success · Started 6:31 AM UTC · Completed 6:48 AM UTC |
ReviewFindingsMedium
Low
Previous runReviewFindingsMedium
Low
Labels: PR modifies review pipeline post-review scripts |
|
/fs-fix |
|
🤖 Finished Fix · ✅ Success · Started 9:52 AM UTC · Completed 9:59 AM UTC |
- Document the assumption that fullsend post-review's exit code does not distinguish "comment posted, review failed" from "nothing posted", and note the HTTP status code limitation (findings 1, 5) - Document intentional omission of gh pr close in the reject fallback as a conservative measure for uncertain state (finding 2) - Skip removing the target outcome label in the fallback's stale-label loop to avoid a pointless unlabel/relabel cycle (finding 7) - Sanitize ACTION for GHA workflow command output in the fallback warning message, matching the existing label-action pattern (finding 8) - Document POST_REVIEW_RETRY_DELAY in docs/review.md Variables table (finding 9) Addresses review feedback on #161
🔧 Fix agent — iteration 1 (human-triggered)Addressed 6 of 9 review findings with code changes: documented degraded-mode assumptions and HTTP status limitation, documented intentional omission of PR close in reject fallback, fixed stale-label removal to skip target label, sanitized ACTION in GHA warning output, and documented POST_REVIEW_RETRY_DELAY. Disagreed with 3 findings (protected-path is informational, scope-creep is a reasonable design choice, code-duplication refactoring is out of scope). Fixed (6):
Disagreed (3):
Tests: passed Updated by fullsend fix agent |
|
/fs-review |
|
🤖 Finished Review · ✅ Success · Started 10:06 AM UTC · Completed 10:20 AM UTC |
waynesun09
left a comment
There was a problem hiding this comment.
Automated 3-agent review pass (2 Claude-based agents + a Grok-based agent), focused on the retry loop's double-posting/idempotency risk, backoff correctness, and premature decisions, per the PR's own stated intent.
2 HIGH findings on the new retry loop: a missing idempotency guard against double-posting the formal review, and a set -euo pipefail interaction that lets an empty/non-numeric POST_REVIEW_RETRY_DELAY crash the script before the degraded-mode fallback runs. 3 MEDIUM findings: two premature-decision items (blanket transient-failure assumption; an overstated CLI-limitation claim tied to issue #159's requirements 2/4) and one test-coverage gap that let the crash bug ship undetected.
Prior review rounds' findings on this PR (label-fallback-without-verification, reject/close omission, label churn, GHA ::warning:: injection, POST_REVIEW_RETRY_DELAY doc gap, protected-path/scope-creep/code-duplication) are already fixed, documented as intentional, or declared out of scope, and are not repeated here.
|
/fs-fix |
|
🤖 Finished Fix · ✅ Success · Started 3:04 PM UTC · Completed 3:14 PM UTC |
- Add idempotency guard: snapshot review ID before retry loop, check
for newly-posted review before each retry to prevent double-posting
if a "failed" attempt succeeded server-side (HIGH finding 1)
- Validate POST_REVIEW_RETRY_DELAY with regex (^[0-9]+$) instead of
${VAR+x} — empty/non-numeric values now fall through to the
progressive default instead of crashing under set -e (HIGH finding 2)
- Capture fullsend stderr per attempt and include first line in
::warning:: messages for diagnostics (MEDIUM finding 4)
- Document retry-all-non-0/non-10 as deliberate trade-off since
fullsend lacks distinct exit codes for failure classes (MEDIUM finding 3)
- Correct degraded-mode comment: the CLI does expose diagnostic output
on stderr; the limitation is no distinct exit codes (MEDIUM finding 4)
- Add 3 test cases exercising default backoff (unset), empty-string,
and non-numeric POST_REVIEW_RETRY_DELAY overrides (MEDIUM finding 5)
Addresses review feedback on #161
🔧 Fix agent — iteration 2 (human-triggered)Addressed all 5 review findings (2 HIGH, 3 MEDIUM). Added idempotency guard for the retry loop, fixed POST_REVIEW_RETRY_DELAY crash on empty/non-numeric values, captured stderr for diagnostics, documented retry-all as deliberate trade-off, corrected CLI limitation comment, and added 3 tests for default backoff and invalid delay values. Fixed (5):
Tests: passed Decision points
Updated by fullsend fix agent |
waynesun09
left a comment
There was a problem hiding this comment.
Recheck pass on commit 84f3128 (previous pass reviewed c39f68b)
All 5 findings from the previous review pass are fixed in the current head and their threads have been resolved:
- Idempotency guard added (snapshot + per-retry check against the reviews API)
POST_REVIEW_RETRY_DELAYempty/non-numeric values now validated via regex instead of crashing- Exit-code retry-taxonomy trade-off now documented explicitly
fullsendCLI stderr now captured and surfaced in warning/error messages; the comment's claim about CLI diagnostics corrected- Three new tests cover unset/empty/non-numeric
POST_REVIEW_RETRY_DELAY
This pass dispatched 4 independent reviewers (2x Claude, Grok, Codex) against the current diff, focused on whether those fixes are actually sound and whether the fix commits introduced anything new. All 4 independently converged on a real gap in the idempotency guard itself (see inline finding below). Findings are posted inline where they land on diffed lines; one documentation finding falls outside this PR's diff hunks and is included here instead:
[MEDIUM] docs/review.md's Control-labels table (the ready-for-merge / requires-manual-review rows, not touched by this PR's diff) doesn't mention that either label can now be applied via the degraded-mode fallback without a confirmed posted review. Worth a follow-up doc note given the fallback-condition finding below — label presence no longer strictly implies a review was successfully posted.
Two single-agent findings below were independently verified against the fullsend CLI source and issue #159's text before inclusion. A third single-agent finding's supporting evidence (an internal retry-count claim) contained a factual error that's corrected in the finding text. Everything below MEDIUM severity was discarded per this pass's verification threshold.
| _current_review_id=$(gh api "repos/${REPO_FULL_NAME}/pulls/${PR_NUMBER}/reviews" \ | ||
| --jq 'map(.id) | max // 0' 2>/dev/null) || true | ||
| _current_review_id="${_current_review_id:-0}" | ||
| if [ "${_current_review_id}" -gt "${_last_review_id}" ]; then |
There was a problem hiding this comment.
[HIGH] Idempotency guard has no author/commit filter and fails open on its own snapshot query
Finding: The new idempotency guard (_last_review_id snapshot at lines 343-345, comparison at lines 379-388) treats any increase in max(.id) across all PR reviews as proof that this run's own fullsend post-review call succeeded server-side — it never filters by posting identity or by commit_id matching the reviewed SHA. Two independent ways this produces a false "already posted" conclusion, both of which set POST_REVIEW_EXIT=0 and fall straight into the unconditional "Outcome labels" section: (1) if a human or another bot submits any review on the PR during the 5-20s backoff window, the guard concludes the failed attempt "actually succeeded"; (2) the initial snapshot swallows gh api failures via 2>/dev/null || true, silently defaulting _last_review_id to 0 — on a PR that already has prior reviews (e.g. from an earlier /fs-review redispatch cycle, which this pipeline explicitly supports), the very next per-retry check will see _current_review_id > 0 and falsely conclude success on the first failed attempt. In both cases the only trace is a low-visibility ::notice::, not a ::warning::/::error:: — the intended review is never posted, yet the workflow reports success and can still attach an outcome label.
Suggestion: Filter the reviews query to this run's own identity (cache gh api user --jq .login once, or resolve the bot login) and/or match on commit_id equal to the reviewed head SHA, e.g. --jq --arg sha "$HEAD_SHA" 'map(select(.commit_id == $sha)) | map(.id) | max // 0'. Treat a failed snapshot query as "unknown" rather than 0, since 0 makes the guard trivially satisfied by any pre-existing review.
Independently flagged by all 4 reviewers in this pass (Claude x2, Grok, Codex); one reproduced the false-positive empirically by simulating a permanent fullsend failure alongside an incrementing (but unrelated) reviews-API mock.
| # codes do not differentiate failure modes.) Applying the label here is a | ||
| # best-effort measure — exit 1 still signals CI failure so a human can | ||
| # verify. | ||
| echo "Attempting degraded-mode label fallback..." |
There was a problem hiding this comment.
[HIGH] Degraded-mode fallback applies the outcome label without confirming the review comment was actually posted
Finding: Issue #159's proposed change and validation criteria both explicitly condition the fallback on the comment having been posted ("if all retries are exhausted but the review comment was already posted successfully, attempt to apply the label"). The implementation drops that condition: the comment block above this line says outright "We cannot distinguish 'comment posted, review failed' from 'nothing posted'... Applying the label here is a best-effort measure" — and then applies it unconditionally whenever all 3 attempts exit non-zero/non-10. Tracing fullsend post-review's actual implementation confirms this matters: it posts the sticky comment first and returns immediately if that call itself fails, before ever attempting the formal review. So a failure mode common to all 3 attempts (e.g. an invalid/expired token, or the API being entirely unreachable) means nothing is posted to the PR — no comment, no review — yet the fallback still applies e.g. ready-for-merge for an approve verdict, purely from the sandboxed agent's proposed JSON action. The only signal anything went wrong is the workflow's exit code, which is easy to miss if downstream automation gates on label presence — the exact failure mode issue #159 was originally filed over.
Suggestion: Before applying any fallback label, verify a sticky comment actually landed — e.g. check the PR's issue comments for the <!-- fullsend:review-agent --> marker the CLI itself uses — and only proceed with the fallback if the marker is present, matching the issue's stated condition.
Flagged by 1 of 4 reviewers, who traced the actual call order and marker constant in the fullsend CLI source; independently confirmed here against the same source and against issue #159's text.
|
|
||
| # Snapshot the latest review ID before the retry loop for idempotency. | ||
| # If a "failed" attempt actually posted a review, the retry detects it. | ||
| _last_review_id=$(gh api "repos/${REPO_FULL_NAME}/pulls/${PR_NUMBER}/reviews" \ |
There was a problem hiding this comment.
[MEDIUM] Reviews-list API calls omit --paginate, silently capped at the first 30 (oldest) reviews
Finding: Both new gh api repos/.../pulls/.../reviews calls (lines 343, 381) fetch only GitHub's default first page — 30 items, oldest-first — unlike the other multi-page listing calls already in this file (labels --paginate, issues/.../comments --paginate). Once a PR accumulates more than 30 reviews over its lifetime (plausible for a heavily re-reviewed/re-dispatched PR under this same pipeline), max(.id) stays pinned to the same stale page-1 value on every check, and the -gt comparison can never fire — silently disabling the idempotency guard on exactly the busy, long-lived PRs most likely to need it.
Suggestion: Add --paginate to both calls, consistent with the existing convention in this file.
Flagged by 3 of 4 reviewers; one reproduced it directly (35 simulated reviews -> unpaginated query returns 30 regardless of the true max).
| fi | ||
| echo "::warning::fullsend post-review attempt ${_pr_attempt}/${POST_REVIEW_MAX_ATTEMPTS} failed (exit ${POST_REVIEW_EXIT}${_pr_err_detail}) — retrying in ${_backoff}s" | ||
| sleep "${_backoff}" | ||
| else |
There was a problem hiding this comment.
[MEDIUM] Idempotency check is never re-run before the final "retries exhausted" branch
Finding: The idempotency re-check only runs inside if [ "${_pr_attempt}" -lt "${POST_REVIEW_MAX_ATTEMPTS}" ] (line 379) — i.e. between attempts 1->2 and 2->3. On the last attempt, a failure falls straight into this else branch ("all retries exhausted") with no re-query, then into the degraded-mode fallback and exit 1. If attempt 3 itself is a false negative — the server processed the write but the CLI reported failure, the exact scenario this whole PR exists to handle — the script has no way to detect it at the one moment it matters most, and reports a failure (re-applying a label via the fallback) for a run that may have actually succeeded.
Suggestion: Run the same review-ID comparison once more immediately after the final attempt fails, before falling into the elif [ "${POST_REVIEW_EXIT}" -ne 0 ] branch, so a genuinely successful last attempt is still reported as success.
Flagged by 3 of 4 reviewers.
| #!/usr/bin/env bash | ||
|
|
||
| # gh api repos/.../pulls/.../reviews (idempotency guard) | ||
| if [[ "\$1" == "api" ]] && [[ "\$2" == *"/reviews"* ]] && [[ "\$*" == *"--jq"* ]]; then |
There was a problem hiding this comment.
[MEDIUM] premature-decision — No test exercises the idempotency guard's actual trigger path
Finding: This mock's reviews-API handler (and its non-retry counterpart earlier in the file) unconditionally echo "0"s regardless of call count. Across all 7 retry tests plus the 3 backoff/validation tests, _current_review_id can therefore never exceed _last_review_id — the actual anti-double-post branch this PR adds in post-review.sh (POST_REVIEW_EXIT=0; break on the idempotency-guard hit) has zero test coverage. This is the same untested-safety-mechanism pattern that let the original crash bug ship unnoticed last round: the guard added specifically to close a HIGH-severity risk is itself shipped with no test proving it fires.
Suggestion: Add a mock reviews handler that returns an incrementing max-id on a later invocation (mirroring the existing MOCK_FULLSEND_COUNTER pattern), and a test asserting the "idempotency guard" notice appears and no further fullsend call happens after it fires.
Flagged by all 4 reviewers.
| # "Error: github api: <status> <message>" on API failures). | ||
| _pr_err_detail="" | ||
| if [ -s "${_pr_stderr}" ]; then | ||
| _pr_err_detail=" — $(head -1 "${_pr_stderr}")" |
There was a problem hiding this comment.
[MEDIUM] Raw fullsend stderr is interpolated into GHA ::warning:: annotations without the sanitization used elsewhere in this file
Finding: _pr_err_detail is built from unsanitized fullsend stderr and embedded directly into the retry ::warning:: messages a few lines below. Elsewhere in this same file, external/derived strings destined for GHA workflow-command output are explicitly stripped of \n, \r, and :: before use — see LA_ACTION/LA_LABEL earlier in the file, and _safe_action (added by this very PR for a sibling message in the same fallback block, just further down). Since GitHub's validation-error bodies can echo back parts of the request that triggered them, a 422 whose message reflects PR-controlled input could plausibly propagate an unsanitized :: sequence into a workflow-command annotation.
Suggestion: Apply the same \n/\r/::-stripping already used for _safe_action to _pr_err_detail before interpolating it into the ::warning:: messages.
Flagged by 1 of 4 reviewers; verified here against this file's own established sanitization pattern.
| # NOTE: fullsend does not expose distinct exit codes for transient vs. | ||
| # permanent failures (e.g. invalid token, malformed result file). As a | ||
| # deliberate trade-off, all non-zero/non-10 codes are retried — a permanent | ||
| # failure wastes at most two retry attempts (~20s) before falling through to |
There was a problem hiding this comment.
[MEDIUM] premature-decision — the "~20s worst case" comment doesn't account for fullsend's own internal HTTP retry/backoff
Finding: This comment states a permanent failure "wastes at most two retry attempts (~20s)." That holds only for failure classes the fullsend CLI doesn't already retry internally. fullsend's own GitHub client retries 429/5xx/secondary-rate-limited-403 responses internally (maxRetries = 3 in its GitHub client — not 5, as initially reported to me and corrected after checking the source directly), and for secondary rate limits specifically uses a 60s-plus-jitter base backoff — so a single fullsend post-review invocation can itself take roughly two minutes to give up before ever returning control to this script. Stacking this shell-level retry loop (3 attempts) on top means the true worst case under sustained rate-limiting is on the order of several minutes, not "~20s" — though 422 (this PR's specific motivating failure) is not retried internally by the Go client, so the common case is unaffected.
Suggestion: Scope the "~20s" claim to failure classes the CLI doesn't already retry (e.g. 422, auth errors), or drop the specific number and note that total worst-case latency depends on fullsend's own internal retry behavior for rate-limit-class failures.
Flagged by 1 of 4 reviewers, who traced the retry constants in the fullsend CLI source; re-verified directly against that source here.
Wrap the fullsend post-review call in a retry loop (max 3 attempts with 5s/15s backoff) to handle transient GitHub API errors such as 422 Unprocessable Entity during PR transitional states.
Previously, a single non-zero exit (other than code 10 for stale-head) caused post-review.sh to exit immediately. This meant a valid review comment could be posted while the formal review API call failed, leaving the PR without an outcome label and requiring human diagnosis.
Changes:
Exit code 10 (stale-head) bypasses retry and uses existing handler.
On exhausted retries, a degraded-mode fallback applies the outcome
label directly so the PR is not left in limbo. The script still
exits 1 to signal the formal review was not posted.
retry-then-succeed, retries-exhausted with fallback for approve/
comment/request-changes actions, stale-head bypass, and retry
log message format.
POST_REVIEW_RETRY_DELAY env var overrides backoff for all retries (set to 0 in tests to skip sleep).
Closes #159
Post-script verification
agent/159-retry-post-review)51561b23f669baf5e6ad027c5de39b0fdb199ab9..HEAD)